From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 11:13:19 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 11:13:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 11:13:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 11:13:20 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:13:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 15:13:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:13:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 13 15:13:53 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:53 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 13 15:13:53 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:13:53 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 15:21:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 15:21:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 15:21:15 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 16:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 16:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 16:00:48 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 20:01:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 13 20:01:38 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 13 20:01:38 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:38 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 13 20:01:38 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 08:02:19 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 08:02:19 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 08:02:19 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 14 08:02:20 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:20 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 14 08:02:20 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 08:02:20 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 12:00:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 12:00:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 12:00:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 14 12:00:45 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:45 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 14 12:00:45 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 12:00:45 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 16:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 16:00:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 16:00:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 16:00:44 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 20:00:41 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 20:00:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 14 20:00:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 14 20:00:42 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 14 20:00:42 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:42 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 14 20:00:42 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 08:02:06 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 08:02:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 08:02:07 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 08:02:07 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:07 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 08:02:07 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 12:00:50 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 12:00:50 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 12:00:50 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 16:00:43 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 16:00:43 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 16:00:43 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 20:00:40 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 20:00:43 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 15 20:00:43 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 15 20:00:43 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:43 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 15 20:00:43 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:43 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 15 20:00:43 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 15 20:00:43 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 08:02:29 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 08:02:29 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 08:02:29 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 12:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 12:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 12:00:48 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:47 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 16:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 16:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 16:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 16 16:00:48 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:48 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 16 16:00:48 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 16:00:48 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 20:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 20:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 16 20:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 16 20:00:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 20:00:43 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 16 20:00:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 16 20:00:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 16 20:00:44 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 17 08:02:31 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 08:02:32 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 08:02:32 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:32 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 08:02:32 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 08:02:32 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 08:02:32 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:32 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 17 08:02:32 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:32 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 17 08:02:32 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 08:02:32 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 12:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 12:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 12:01:01 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 16:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 16:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 16:00:48 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 17 20:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 17 20:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 17 20:00:51 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 08:04:15 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 08:04:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 08:04:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Jan 18 08:04:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 08:04:15 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:15 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Jan 18 08:04:15 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 08:04:16 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 08:04:16 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:16 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 08:04:16 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 12:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 12:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 12:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 12:01:01 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:26 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 16:01:27 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 16:01:27 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:27 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 16:01:27 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 18 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 18 20:00:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 18 20:00:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 18 20:00:54 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 08:04:36 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 08:04:37 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 08:04:37 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 08:04:37 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 12:00:58 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 12:00:59 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 12:00:59 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 12:00:59 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 16:02:51 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 16:02:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 16:02:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Jan 19 16:02:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 16:02:52 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:52 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Jan 19 16:02:52 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:52 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 16:02:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 16:02:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Jan 19 16:02:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 16:02:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 16:02:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 16:02:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 16:02:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 16:02:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Jan 19 16:02:53 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:53 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Jan 19 16:02:53 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 16:02:53 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 20:00:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 19 20:00:49 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 19 20:00:49 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:49 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Jan 19 20:00:49 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:49 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Jan 19 20:00:49 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 19 20:00:49 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 08:06:42 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 08:06:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 08:06:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 20 08:06:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 08:06:45 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:45 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 08:06:46 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 08:06:46 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:46 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 08:06:46 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 12:02:00 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 12:02:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 12:02:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 20 12:02:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 12:02:00 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:00 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 12:02:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 12:02:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:01 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 20 12:02:02 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 12:02:02 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 16:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 16:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 16:00:51 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 20:00:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 20 20:00:46 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 20 20:00:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 20 20:00:47 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:47 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 20 20:00:47 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 20 20:00:47 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 08:03:50 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 08:03:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 08:03:52 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 08:03:52 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:52 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 08:03:52 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 12:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 21 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 12:01:01 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:01 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 21 12:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 12:01:02 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 12:01:02 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:02 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 12:01:02 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 16:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 16:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 16:00:51 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 20:00:49 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:49 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 20:00:49 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:49 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 21 20:00:50 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 21 20:00:50 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:50 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 21 20:00:50 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 08:02:43 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 08:02:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 08:02:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 08:02:44 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 12:00:50 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 12:00:50 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 12:00:50 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 16:01:19 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 16:01:19 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 16:01:19 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 22 16:01:20 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:20 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 22 16:01:20 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 16:01:20 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 20:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 20:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 22 20:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 22 20:01:14 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 22 20:01:14 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:14 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 22 20:01:14 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 08:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 08:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 08:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 23 08:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 08:02:41 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 08:02:41 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:41 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 08:02:41 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 12:00:55 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:55 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 12:00:56 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 12:00:56 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:56 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 12:00:56 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 16:01:43 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:43 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 16:01:44 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 16:01:45 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:45 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 16:01:45 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 16:01:45 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 16:01:45 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:45 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 23 16:01:45 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:45 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 23 16:01:45 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 16:01:45 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 23 20:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 23 20:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:51 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:51 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 23 20:00:52 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 23 20:00:52 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 08:03:06 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 08:03:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 08:03:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 24 08:03:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 08:03:06 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:07 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 24 08:03:07 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:07 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 08:03:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 08:03:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 08:03:08 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 12:02:50 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 12:02:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 12:02:50 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 24 12:02:51 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 12:02:51 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:51 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 24 12:02:51 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:51 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 12:02:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 12:02:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:52 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 24 12:02:52 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 12:02:52 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 12:02:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 12:02:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 12:02:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 12:02:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 24 12:02:53 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:53 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 24 12:02:53 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 12:02:53 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 16:01:19 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 16:01:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 16:01:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 16:01:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 16:01:20 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 20:01:53 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 24 20:01:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 24 20:01:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 24 20:01:54 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 08:02:00 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 08:02:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 08:02:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 08:02:01 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 12:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 12:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 12:01:01 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 16:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 16:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 16:01:00 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 20:00:52 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Jan 25 20:00:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Jan 25 20:00:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Jan 25 20:00:53 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 08:05:38 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:38 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 08:05:39 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 08:05:39 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:39 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Jan 26 08:05:39 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:40 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Jan 26 08:05:40 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 08:05:40 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 12:01:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 12:01:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 12:01:05 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 16:00:52 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 16:00:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 16:00:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Jan 26 16:00:52 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 16:00:52 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 16:00:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 16:00:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 16:00:53 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 20:00:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Jan 26 20:00:46 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Jan 26 20:00:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Jan 26 20:00:47 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:47 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Jan 26 20:00:47 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Jan 26 20:00:47 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 08:06:34 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:34 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 08:06:34 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:34 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 08:06:34 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 27 08:06:34 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 08:06:35 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:35 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 27 08:06:35 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:35 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 08:06:35 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:36 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 08:06:36 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:36 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 27 08:06:36 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 08:06:36 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 08:06:36 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:36 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 08:06:36 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 08:06:36 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 08:06:36 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:36 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 27 08:06:36 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:37 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 27 08:06:37 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 08:06:37 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 12:02:08 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 12:02:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 12:02:10 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 12:02:10 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:10 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 27 12:02:10 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:10 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 27 12:02:10 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 12:02:10 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 16:01:02 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 27 16:01:06 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 16:01:06 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 16:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 16:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 16:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 16:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 27 16:01:06 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:06 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 27 16:01:06 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 16:01:06 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Jan 27 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Jan 27 20:00:55 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Jan 27 20:00:55 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:55 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Jan 27 20:00:55 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 08:05:34 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:34 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 08:05:34 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 08:05:35 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 08:05:35 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 08:05:35 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:36 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 28 08:05:36 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:36 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 28 08:05:36 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 08:05:36 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 12:01:53 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 12:01:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 12:01:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 12:01:54 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 16:00:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 16:00:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 16:00:54 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Jan 28 20:00:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 20:00:56 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 20:00:56 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:56 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 20:00:56 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Jan 28 20:00:56 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Jan 28 20:00:56 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:56 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Jan 28 20:00:56 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:56 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Jan 28 20:00:56 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Jan 28 20:00:56 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 29 08:07:28 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 08:07:29 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 08:07:29 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:29 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 08:07:29 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 08:07:29 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 08:07:29 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:29 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 29 08:07:29 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:29 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 29 08:07:29 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 08:07:29 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:00:59 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 29 12:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 12:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 12:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 12:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 12:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 12:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 29 12:01:00 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:01:00 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 29 12:01:00 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 12:01:00 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 16:00:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 16:00:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 16:00:53 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 20:00:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Jan 29 20:00:55 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Jan 29 20:00:55 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:55 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Jan 29 20:00:55 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 08:08:36 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 08:08:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:38 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 30 08:08:38 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 08:08:38 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 08:08:38 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:38 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 08:08:38 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 08:08:38 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 08:08:39 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:39 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 30 08:08:39 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:39 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 30 08:08:39 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 08:08:39 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 12:03:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 12:03:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:53 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 30 12:03:53 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:54 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 30 12:03:54 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 12:03:54 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 16:04:06 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 16:04:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 16:04:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 30 16:04:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:07 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 16:04:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 16:04:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 16:04:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 30 16:04:08 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:08 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 30 16:04:08 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 16:04:08 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 20:02:10 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:11 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Jan 30 20:02:11 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 20:02:11 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 20:02:11 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:11 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 20:02:11 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Jan 30 20:02:11 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Jan 30 20:02:11 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:11 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Jan 30 20:02:12 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:12 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Jan 30 20:02:12 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Jan 30 20:02:12 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 08:07:47 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 08:07:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 08:07:48 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:49 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 08:07:49 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 08:07:49 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 08:07:49 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:49 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 31 08:07:49 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:49 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 31 08:07:49 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 08:07:49 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:14 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 12:04:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 12:04:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 12:04:15 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 16:02:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 16:02:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 31 16:02:15 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 16:02:16 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 20:01:07 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 20:01:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 20:01:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Jan 31 20:01:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Jan 31 20:01:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Jan 31 20:01:08 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:08 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Jan 31 20:01:08 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Jan 31 20:01:08 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 08:09:31 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 08:09:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:32 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 08:09:32 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 1 08:09:32 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 08:09:32 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:32 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 1 08:09:32 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:32 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 08:09:32 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:33 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 08:09:33 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:33 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 1 08:09:33 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 08:09:33 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 08:09:33 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:33 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 08:09:33 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 08:09:33 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 08:09:33 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:33 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 1 08:09:34 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:34 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 1 08:09:34 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 08:09:34 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 1 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 12:02:41 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 12:02:42 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 12:02:42 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:42 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 12:02:42 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 16:01:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 16:01:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 16:01:05 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 1 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 1 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 20:00:59 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:00:59 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 1 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 1 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 1 20:01:00 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 08:12:18 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 08:12:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 08:12:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 08:12:21 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 08:12:22 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:22 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 08:12:22 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 08:12:22 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 08:12:22 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:22 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Feb 2 08:12:22 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:22 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Feb 2 08:12:22 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 08:12:22 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 12:02:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 12:02:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 12:02:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 12:02:47 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:10 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Feb 2 16:01:11 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 16:01:11 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 16:01:11 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:11 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 16:01:11 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 16:01:11 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 16:01:11 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:11 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Feb 2 16:01:11 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:11 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Feb 2 16:01:11 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 16:01:11 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 2 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 2 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 2 20:01:00 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 08:11:56 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:11:57 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 08:11:57 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:11:58 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 08:11:58 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Feb 3 08:11:58 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 08:11:58 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:11:59 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Feb 3 08:11:59 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:11:59 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 08:11:59 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:12:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 08:12:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:12:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Feb 3 08:12:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 08:12:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 08:12:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:12:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 08:12:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 08:12:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 08:12:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:12:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Feb 3 08:12:01 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:12:01 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Feb 3 08:12:01 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 08:12:01 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Feb 3 12:02:38 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 12:02:39 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 12:02:39 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:39 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 12:02:39 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 12:02:39 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 12:02:39 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:39 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Feb 3 12:02:39 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:39 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Feb 3 12:02:39 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 12:02:39 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 16:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 16:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 16:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 16:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:01 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 16:01:01 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 20:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 3 20:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 3 20:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 3 20:01:06 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 08:07:45 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Feb 4 08:07:47 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 08:07:47 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 08:07:47 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:47 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 08:07:47 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 08:07:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 08:07:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:47 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Feb 4 08:07:47 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:47 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Feb 4 08:07:47 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 08:07:47 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Feb 4 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 12:02:41 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:41 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Feb 4 12:02:41 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 12:02:42 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 12:02:42 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:42 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 12:02:42 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 16:01:12 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 16:01:12 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 16:01:12 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 4 20:01:02 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 4 20:01:02 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 4 20:01:02 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 08:07:54 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 08:07:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 08:07:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:54 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Feb 5 08:07:55 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:55 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Feb 5 08:07:55 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 08:07:55 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 12:02:41 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 12:02:41 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:41 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:41 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Feb 5 12:02:42 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 12:02:42 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 16:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 16:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 16:01:06 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:58 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Feb 5 20:00:59 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 20:00:59 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 20:00:59 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:59 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 20:00:59 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 5 20:00:59 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 5 20:00:59 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:59 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Feb 5 20:00:59 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:59 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Feb 5 20:00:59 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 5 20:00:59 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 08:14:28 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 08:14:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 08:14:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Feb 6 08:14:32 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 08:14:32 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:32 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Feb 6 08:14:32 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:33 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 08:14:33 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:33 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 08:14:33 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:33 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Feb 6 08:14:35 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 08:14:35 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 08:14:35 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:35 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 08:14:35 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 08:14:36 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 08:14:37 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:37 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Feb 6 08:14:37 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:38 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Feb 6 08:14:38 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 08:14:38 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 12:02:39 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 12:02:40 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 12:02:40 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 12:02:40 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 16:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 16:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 16:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 16:01:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 16:01:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 16:01:05 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 6 20:00:59 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 6 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 6 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 6 20:01:00 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 08:22:44 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 08:22:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 08:22:47 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Feb 7 08:22:48 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 08:22:48 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:48 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Feb 7 08:22:49 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:49 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 08:22:49 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:49 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 08:22:49 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:51 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Feb 7 08:22:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 08:22:51 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 08:22:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 08:22:52 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 08:22:52 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 08:22:52 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:52 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Feb 7 08:22:52 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:53 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Feb 7 08:22:53 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 08:22:53 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 12:03:15 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 12:03:16 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 12:03:16 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 12:03:16 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 16:01:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 16:01:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 16:01:08 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 7 20:01:03 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 7 20:01:03 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 7 20:01:03 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 08:21:59 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 08:22:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 08:22:05 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 8 08:22:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 08:22:06 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:06 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 8 08:22:06 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:06 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 08:22:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 08:22:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 8 08:22:07 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 08:22:07 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 08:22:07 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:07 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 08:22:07 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 08:22:07 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 08:22:07 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:07 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 8 08:22:07 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:08 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 8 08:22:08 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 08:22:08 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 12:03:19 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 12:03:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 12:03:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 8 12:03:19 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 12:03:20 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:20 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 8 12:03:20 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:20 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 12:03:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 12:03:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 8 12:03:20 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 12:03:21 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 12:03:21 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:21 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 12:03:21 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 12:03:21 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 12:03:21 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:21 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 8 12:03:21 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:21 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 8 12:03:21 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 12:03:21 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:13 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 8 16:01:14 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 16:01:14 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 16:01:14 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:14 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 16:01:14 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 16:01:14 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 16:01:14 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:14 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 8 16:01:14 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:14 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 8 16:01:14 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 16:01:14 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 8 20:01:04 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 8 20:01:04 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 8 20:01:04 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 08:09:05 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 08:09:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 08:09:06 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Feb 9 08:09:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 08:09:07 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:07 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Feb 9 08:09:07 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:07 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 08:09:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 08:09:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Feb 9 08:09:08 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 08:09:08 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 08:09:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 08:09:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 08:09:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 08:09:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Feb 9 08:09:09 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:09 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Feb 9 08:09:09 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 08:09:09 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 12:03:03 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 12:03:04 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 12:03:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 12:03:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 12:03:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Feb 9 12:03:06 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:06 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Feb 9 12:03:06 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 12:03:06 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 16:01:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 16:01:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 16:01:15 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 20:01:07 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:07 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 9 20:01:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 9 20:01:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 9 20:01:09 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 08:19:26 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:27 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 08:19:27 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:27 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 08:19:27 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Feb 10 08:19:27 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 08:19:28 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:28 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Feb 10 08:19:28 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:28 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 08:19:28 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:28 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 08:19:29 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:30 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Feb 10 08:19:30 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 08:19:30 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 08:19:30 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:30 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 08:19:30 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 08:19:30 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 08:19:31 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:31 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Feb 10 08:19:31 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:31 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Feb 10 08:19:32 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 08:19:32 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 12:03:35 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:35 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 12:03:35 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:35 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 12:03:35 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 12:03:36 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 12:03:36 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 12:03:36 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:36 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Feb 10 12:03:37 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:37 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Feb 10 12:03:37 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 12:03:37 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Feb 10 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 16:01:08 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:08 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 16:01:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 16:01:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 16:01:09 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Feb 10 20:01:12 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Feb 10 20:01:12 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Feb 10 20:01:12 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 08:12:43 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 08:12:44 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 08:12:45 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 08:12:45 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:45 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 08:12:45 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 12:02:37 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 12:02:37 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 12:02:37 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 16:01:03 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 16:01:03 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 16:01:04 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:04 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Feb 11 16:01:04 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:04 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Feb 11 16:01:04 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 16:01:04 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 20:00:56 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sat Feb 11 20:00:56 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sat Feb 11 20:00:57 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:57 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sat Feb 11 20:00:57 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:57 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sat Feb 11 20:00:57 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sat Feb 11 20:00:57 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 08:09:10 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:11 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 08:09:11 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:11 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 08:09:11 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Feb 12 08:09:11 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 08:09:11 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:11 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Feb 12 08:09:11 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:11 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 08:09:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 08:09:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Feb 12 08:09:12 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 08:09:13 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 08:09:13 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:13 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 08:09:13 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 08:09:13 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 08:09:14 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:14 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Feb 12 08:09:14 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:14 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Feb 12 08:09:14 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 08:09:14 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 12:02:44 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 12:02:45 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 12:02:45 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 12:02:45 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:46 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Feb 12 12:02:46 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:46 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Feb 12 12:02:46 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 12:02:46 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Feb 12 16:01:08 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 16:01:09 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 16:01:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 16:01:09 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 16:01:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 16:01:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Feb 12 16:01:09 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:09 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Feb 12 16:01:09 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 16:01:09 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Sun Feb 12 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Sun Feb 12 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Sun Feb 12 20:01:00 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 08:16:43 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 08:16:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 08:16:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Feb 13 08:16:45 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 08:16:45 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:45 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Feb 13 08:16:45 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 08:16:46 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 08:16:46 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 08:16:46 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:46 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Feb 13 08:16:47 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:47 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Feb 13 08:16:47 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 08:16:47 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 12:03:03 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 12:03:04 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 12:03:04 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:04 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 12:03:04 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 16:01:23 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 16:01:23 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 16:01:23 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 20:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:05 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Mon Feb 13 20:01:05 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 20:01:06 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 20:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 20:01:06 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Mon Feb 13 20:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Mon Feb 13 20:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:06 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Mon Feb 13 20:01:06 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:06 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Mon Feb 13 20:01:06 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Mon Feb 13 20:01:06 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 08:19:02 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 08:19:03 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 08:19:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:04 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Feb 14 08:19:04 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 08:19:04 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 08:19:04 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 08:19:05 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 08:19:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 08:19:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:05 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Feb 14 08:19:05 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:05 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Feb 14 08:19:05 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 08:19:05 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 12:03:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 12:03:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:20 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Feb 14 12:03:20 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:21 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Feb 14 12:03:21 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 12:03:21 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 16:01:31 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 16:01:31 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 16:01:31 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 20:01:11 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Tue Feb 14 20:01:12 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Tue Feb 14 20:01:12 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Tue Feb 14 20:01:12 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 08:27:00 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 08:27:02 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:02 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 08:27:02 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 15 08:27:04 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 08:27:04 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:04 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 15 08:27:04 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:04 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 08:27:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:06 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 08:27:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:07 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 15 08:27:07 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 08:27:08 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 08:27:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 08:27:08 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 08:27:08 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 08:27:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:09 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 15 08:27:09 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:10 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 15 08:27:10 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 08:27:10 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 12:05:22 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 12:05:23 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 12:05:23 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 12:05:23 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 16:01:30 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 16:01:30 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 16:01:30 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:30 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 15 16:01:31 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:31 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 15 16:01:31 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 16:01:31 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:01 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 19:28:02 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 19:28:02 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:02 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 19:28:02 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:20 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 20:13:21 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:21 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 20:13:21 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:21 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Wed Feb 15 20:13:21 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 20:13:21 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 20:13:22 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:22 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 20:13:22 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Wed Feb 15 20:13:22 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Wed Feb 15 20:13:22 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:22 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Wed Feb 15 20:13:22 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:22 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Wed Feb 15 20:13:22 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Wed Feb 15 20:13:22 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 16 08:45:53 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:55 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 16 08:45:55 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:55 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Thu Feb 16 08:45:55 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Thu Feb 16 08:45:55 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 16 08:45:55 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:55 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Thu Feb 16 08:45:56 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:56 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 16 08:45:57 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:57 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 16 08:45:57 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:57 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Thu Feb 16 08:45:57 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 16 08:45:58 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 16 08:45:58 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:58 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 16 08:45:59 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Thu Feb 16 08:45:59 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Thu Feb 16 08:45:59 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:59 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Thu Feb 16 08:45:59 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:59 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Thu Feb 16 08:45:59 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Thu Feb 16 08:45:59 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 11:11:02 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] Copy ing rows from tables... Message-ID: <1065773461.7986.18.camel@jerry.westrick.local> Dear sirs. I have the situation where I "replicate" a database table. That is I have a complete copy of a (MS SQL) database table stored locally (in Postgress). The system has been setup so that I can identify the modified rows. Theese I need to retrieve and store (either insert or update) into my postgress table. I access both databases via your zope mxodbc product. My question is, which is the easiest way to retrive a row from the MS-SQL connection, and inseert/update itno the postgress connection. THe standard zope solution, requires the listing of all field names in both the update and insert statements. This is terribly maintenance prone. Since the 2 tables are identical (except my version has an additional field which is updated via trigger), I suspect that there must be a lower level routines, which i can use which takes advantage of the fact that the 2 tables are similarly structured. Thanking you in advance for your support. Jerry Westrick. From mal at lemburg.com Fri Oct 10 11:41:49 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065773461.7986.18.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> Message-ID: <3F8670CD.1000400@lemburg.com> Jerry Westrick wrote: > Dear sirs. > > I have the situation where I "replicate" a database table. That is I > have a complete copy of a (MS SQL) database table stored locally (in > Postgress). The system has been setup so that I can identify the > modified rows. Theese I need to retrieve and store (either insert or > update) into my postgress table. > > I access both databases via your zope mxodbc product. > > My question is, which is the easiest way to retrive a row from > the MS-SQL connection, and inseert/update itno the postgress connection. > > THe standard zope solution, requires the listing of all field names in > both the update and insert statements. This is terribly maintenance > prone. Since the 2 tables are identical (except my version has an > additional field which is updated via trigger), I suspect that there > must be a lower level routines, which i can use which takes advantage > of the fact that the 2 tables are similarly structured. > > Thanking you in advance for your support. It is quite easy to fetch the field names for both tables and format an SQL query for the update from that list. A typical way of doing that is to run the query "SELECT * FROM MyTable WHERE 1=0" and then having a look at the result set which is empty but does list the field names. Another approach is to use the .columns() method on the mxODBC Zope DA connection object: def columns(self, table_name): """ Returns a list of dictionaries with entries 'Name', 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') for each column in the table table_name. """ -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 12:04:00 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F8670CD.1000400@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> Message-ID: <1065776640.7985.36.camel@jerry.westrick.local> Mr. Lemburg First thansk for the prompt reply... (it's nice to be supported out of europe every once in a while)! Looking in your documentation, is see a routne called "execute" which I could use to avoid having to do SQL formating. So following your suggestion, and documentation I can: 1- Get list of columns (in correct order from result set) 2- build insert/update stmt with '?' parameter markers 3- then for each row returned from original select pass the resulting row as 'params' parameter to "execute" method I wonder if the returned row would be acceptable as input to the params parameter.... Jerry Westrick On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Dear sirs. > > > > I have the situation where I "replicate" a database table. That is I > > have a complete copy of a (MS SQL) database table stored locally (in > > Postgress). The system has been setup so that I can identify the > > modified rows. Theese I need to retrieve and store (either insert or > > update) into my postgress table. > > > > I access both databases via your zope mxodbc product. > > > > My question is, which is the easiest way to retrive a row from > > the MS-SQL connection, and inseert/update itno the postgress connection. > > > > THe standard zope solution, requires the listing of all field names in > > both the update and insert statements. This is terribly maintenance > > prone. Since the 2 tables are identical (except my version has an > > additional field which is updated via trigger), I suspect that there > > must be a lower level routines, which i can use which takes advantage > > of the fact that the 2 tables are similarly structured. > > > > Thanking you in advance for your support. > > It is quite easy to fetch the field names for both tables and > format an SQL query for the update from that list. > > A typical way of doing that is to run the query "SELECT * FROM > MyTable WHERE 1=0" and then having a look at the result set > which is empty but does list the field names. > > Another approach is to use the .columns() method on the mxODBC > Zope DA connection object: > > def columns(self, table_name): > > """ Returns a list of dictionaries with entries 'Name', > 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > for each column in the table table_name. > > """ From mal at lemburg.com Fri Oct 10 12:38:41 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065776640.7985.36.camel@jerry.westrick.local> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> Message-ID: <3F867E21.80004@lemburg.com> Jerry Westrick wrote: > Mr. Lemburg > > First thansk for the prompt reply... > (it's nice to be supported out of europe every once in a while)! > > Looking in your documentation, is see a routne called "execute" > which I could use to avoid having to do SQL formating. > > So following your suggestion, and documentation I can: > 1- Get list of columns (in correct order from result set) > 2- build insert/update stmt with '?' parameter markers > 3- then for each row returned from original select > pass the resulting row as 'params' parameter to "execute" method Not quite: the '?' marker is usually only allowed in the WHERE clause (plus some other places which vary by ODBC driver). You should get the list of columns and then format them directly into the SQL: "SELECT %s WHERE ... " % (','.join(columnnames),) will get you there. > I wonder if the returned row would be acceptable as input to the > params parameter.... > > > Jerry Westrick > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > >>Jerry Westrick wrote: >> >>>Dear sirs. >>> >>>I have the situation where I "replicate" a database table. That is I >>>have a complete copy of a (MS SQL) database table stored locally (in >>>Postgress). The system has been setup so that I can identify the >>>modified rows. Theese I need to retrieve and store (either insert or >>>update) into my postgress table. >>> >>>I access both databases via your zope mxodbc product. >>> >>>My question is, which is the easiest way to retrive a row from >>>the MS-SQL connection, and inseert/update itno the postgress connection. >>> >>>THe standard zope solution, requires the listing of all field names in >>>both the update and insert statements. This is terribly maintenance >>>prone. Since the 2 tables are identical (except my version has an >>>additional field which is updated via trigger), I suspect that there >>>must be a lower level routines, which i can use which takes advantage >>>of the fact that the 2 tables are similarly structured. >>> >>>Thanking you in advance for your support. >> >>It is quite easy to fetch the field names for both tables and >>format an SQL query for the update from that list. >> >>A typical way of doing that is to run the query "SELECT * FROM >>MyTable WHERE 1=0" and then having a look at the result set >>which is empty but does list the field names. >> >>Another approach is to use the .columns() method on the mxODBC >>Zope DA connection object: >> >> def columns(self, table_name): >> >> """ Returns a list of dictionaries with entries 'Name', >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') >> for each column in the table table_name. >> >> """ > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Jerry at Westrick.Com Fri Oct 10 13:01:04 2003 From: Jerry at Westrick.Com (Jerry Westrick) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <3F867E21.80004@lemburg.com> References: <1065773461.7986.18.camel@jerry.westrick.local> <3F8670CD.1000400@lemburg.com> <1065776640.7985.36.camel@jerry.westrick.local> <3F867E21.80004@lemburg.com> Message-ID: <1065780064.7986.39.camel@jerry.westrick.local> Okay, thanks... I see I'll have to do SQL formatting myself, and will have to use the column descriptors to decide on how to do so... That answers my questions... Thanks for you time! Jerry On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > Jerry Westrick wrote: > > Mr. Lemburg > > > > First thansk for the prompt reply... > > (it's nice to be supported out of europe every once in a while)! > > > > Looking in your documentation, is see a routne called "execute" > > which I could use to avoid having to do SQL formating. > > > > So following your suggestion, and documentation I can: > > 1- Get list of columns (in correct order from result set) > > 2- build insert/update stmt with '?' parameter markers > > 3- then for each row returned from original select > > pass the resulting row as 'params' parameter to "execute" method > > Not quite: the '?' marker is usually only allowed in the > WHERE clause (plus some other places which vary by ODBC driver). > You should get the list of columns and then format them > directly into the SQL: > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > will get you there. > > > I wonder if the returned row would be acceptable as input to the > > params parameter.... > > > > > > Jerry Westrick > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > >>Jerry Westrick wrote: > >> > >>>Dear sirs. > >>> > >>>I have the situation where I "replicate" a database table. That is I > >>>have a complete copy of a (MS SQL) database table stored locally (in > >>>Postgress). The system has been setup so that I can identify the > >>>modified rows. Theese I need to retrieve and store (either insert or > >>>update) into my postgress table. > >>> > >>>I access both databases via your zope mxodbc product. > >>> > >>>My question is, which is the easiest way to retrive a row from > >>>the MS-SQL connection, and inseert/update itno the postgress connection. > >>> > >>>THe standard zope solution, requires the listing of all field names in > >>>both the update and insert statements. This is terribly maintenance > >>>prone. Since the 2 tables are identical (except my version has an > >>>additional field which is updated via trigger), I suspect that there > >>>must be a lower level routines, which i can use which takes advantage > >>>of the fact that the 2 tables are similarly structured. > >>> > >>>Thanking you in advance for your support. > >> > >>It is quite easy to fetch the field names for both tables and > >>format an SQL query for the update from that list. > >> > >>A typical way of doing that is to run the query "SELECT * FROM > >>MyTable WHERE 1=0" and then having a look at the result set > >>which is empty but does list the field names. > >> > >>Another approach is to use the .columns() method on the mxODBC > >>Zope DA connection object: > >> > >> def columns(self, table_name): > >> > >> """ Returns a list of dictionaries with entries 'Name', > >> 'Type', 'Precision', 'Scale', 'Nullable' ('with Null' or '') > >> for each column in the table table_name. > >> > >> """ > > > > > > > > _______________________________________________________________________ > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users From sholden at holdenweb.com Sun Oct 12 15:10:34 2003 From: sholden at holdenweb.com (Steve Holden) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] Copy ing rows from tables... In-Reply-To: <1065780064.7986.39.camel@jerry.westrick.local> Message-ID: Might I immodestly suggest looking at the recipe I wrote for the Python cookbook? This demonstrates how the description can be used to access aspects of the retrieved rows. See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81189 and possibly http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Databases for further hopefully useful information. regards -- Steve Holden +1 703 278 8281 http://www.holdenweb.com/ Improve the Internet http://vancouver-webpages.com/CacheNow/ Python Web Programming http://pydish.holdenweb.com/pwp/ Interview with GvR August 14, 2003 http://www.onlamp.com/python/ > -----Original Message----- > From: egenix-users-bounces@lists.egenix.com > [mailto:egenix-users-bounces@lists.egenix.com]On Behalf Of Jerry > Westrick > Sent: Friday, October 10, 2003 6:01 AM > To: Egenix-Users@Lists.Egenix.Com > Subject: Re: [egenix-users] Copy ing rows from tables... > > > Okay, thanks... > > I see I'll have to do SQL formatting myself, > and will have to use the column descriptors to > decide on how to do so... > > That answers my questions... > > Thanks for you time! > > > Jerry > > > On Fri, 2003-10-10 at 11:38, M.-A. Lemburg wrote: > > Jerry Westrick wrote: > > > Mr. Lemburg > > > > > > First thansk for the prompt reply... > > > (it's nice to be supported out of europe every once in a while)! > > > > > > Looking in your documentation, is see a routne called "execute" > > > which I could use to avoid having to do SQL formating. > > > > > > So following your suggestion, and documentation I can: > > > 1- Get list of columns (in correct order from result set) > > > 2- build insert/update stmt with '?' parameter markers > > > 3- then for each row returned from original select > > > pass the resulting row as 'params' parameter to > "execute" method > > > > Not quite: the '?' marker is usually only allowed in the > > WHERE clause (plus some other places which vary by ODBC driver). > > You should get the list of columns and then format them > > directly into the SQL: > > > > "SELECT %s WHERE ... " % (','.join(columnnames),) > > > > will get you there. > > > > > I wonder if the returned row would be acceptable as input to the > > > params parameter.... > > > > > > > > > Jerry Westrick > > > > > > > > > > > > On Fri, 2003-10-10 at 10:41, M.-A. Lemburg wrote: > > > > > >>Jerry Westrick wrote: > > >> > > >>>Dear sirs. > > >>> > > >>>I have the situation where I "replicate" a database > table. That is I > > >>>have a complete copy of a (MS SQL) database table stored > locally (in > > >>>Postgress). The system has been setup so that I can identify the > > >>>modified rows. Theese I need to retrieve and store > (either insert or > > >>>update) into my postgress table. > > >>> > > >>>I access both databases via your zope mxodbc product. > > >>> > > >>>My question is, which is the easiest way to retrive a row from > > >>>the MS-SQL connection, and inseert/update itno the > postgress connection. > > >>> > > >>>THe standard zope solution, requires the listing of all > field names in > > >>>both the update and insert statements. This is terribly > maintenance > > >>>prone. Since the 2 tables are identical (except my > version has an > > >>>additional field which is updated via trigger), I > suspect that there > > >>>must be a lower level routines, which i can use which > takes advantage > > >>>of the fact that the 2 tables are similarly structured. > > >>> > > >>>Thanking you in advance for your support. > > >> > > >>It is quite easy to fetch the field names for both tables and > > >>format an SQL query for the update from that list. > > >> > > >>A typical way of doing that is to run the query "SELECT * FROM > > >>MyTable WHERE 1=0" and then having a look at the result set > > >>which is empty but does list the field names. > > >> > > >>Another approach is to use the .columns() method on the mxODBC > > >>Zope DA connection object: > > >> > > >> def columns(self, table_name): > > >> > > >> """ Returns a list of dictionaries with entries 'Name', > > >> 'Type', 'Precision', 'Scale', 'Nullable' > ('with Null' or '') > > >> for each column in the table table_name. > > >> > > >> """ > > > > > > > > > > > > > ______________________________________________________________ > _________ > > > eGenix.com User Mailing List http://www.egenix.com/ > > http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Tue Oct 14 17:01:51 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] sql without zsql Message-ID: Without using a ZSQL method, can I get a usable recordset in the form of the recordset a ZSQL method returns? Right now I'm trying to work with the unwieldy data structure returned by the mxODBC "query()" method. It looks something like ([column details], [data]). I would like to get my hands on a recordset object that allows me to loop over the recordset and use dot notation to access each field's value by name. Nathaniel Wingfield From mal at lemburg.com Wed Oct 15 00:13:52 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] sql without zsql In-Reply-To: References: Message-ID: <3F8C6710.9050907@lemburg.com> nwingfield@che-llp.com wrote: > Without using a ZSQL method, can I get a usable recordset in the form of > the recordset a ZSQL method returns? Right now I'm trying to work with the > unwieldy data structure returned by the mxODBC "query()" method. It looks > something like ([column details], [data]). I would like to get my hands on > a recordset object that allows me to loop over the recordset and use dot > notation to access each field's value by name. Try wrapping the results from .query() into a Zope Results object: from Shared.DC.ZRDB.Results import Results rs = Results(conn.query('...')) The same also works for .execute() and .executemany(). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From harald.ragger at icomedias.com Sun Oct 19 15:58:43 2003 From: harald.ragger at icomedias.com (Harald Ragger) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] mxDateTime segfaults... Message-ID: <3F928A83.7010200@icomedias.com> Hello, gentoo linux mxBase 2.0.5 python2.3.2 mxBase builds and installs fine, but segfaults on import: ------------ (gdb) run Starting program: /usr/local/bin/python2.3 [New Thread 16384 (LWP 1807)] Python 2.3.2 (#1, Oct 16 2003, 21:49:53) [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from mx.DateTime import * Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 1807)] mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, second=0) at mx/DateTime/mxDateTime/mxDateTime.c:333 333 _Py_NewReference(datetime); (gdb) [5]+ Stopped gdb python2.3 -------------------- any ideas? i also tried 2.1.0: segfaults too. Thanks, harry From mal at lemburg.com Sun Oct 19 16:06:15 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] mxDateTime segfaults... In-Reply-To: <3F928A83.7010200@icomedias.com> References: <3F928A83.7010200@icomedias.com> Message-ID: <3F928C47.7040408@lemburg.com> Harald Ragger wrote: > Hello, > > gentoo linux > mxBase 2.0.5 > python2.3.2 > > mxBase builds and installs fine, but segfaults on import: > > ------------ > (gdb) run > Starting program: /usr/local/bin/python2.3 > [New Thread 16384 (LWP 1807)] > Python 2.3.2 (#1, Oct 16 2003, 21:49:53) > [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from mx.DateTime import * > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 16384 (LWP 1807)] > mxDateTime_FromDateAndTime (year=1, month=1, day=1, hour=1, minute=1, > second=0) > at mx/DateTime/mxDateTime/mxDateTime.c:333 > 333 _Py_NewReference(datetime); > (gdb) > [5]+ Stopped gdb python2.3 > -------------------- > > any ideas? > > i also tried 2.1.0: segfaults too. This is a known problem, but only occurrs on Python debug builds, not the normal non-debug builds. Solution: use a standard Python build or install the latest egenix-mx-base 2.1 snapshot (see mailing list archives). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Tue Oct 21 18:22:48 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! Message-ID: Hello, after seeing the same case described in the mailing list archives from september, I decided to post my problem too, because it seems really similar Reminder : nwingfield at che-llp.com wrote : "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two or more people run the same query at the same time. This trouble is easily reproduced by opening two browser windows and loading a page that executes a lengthy SQL query in both windows simultaneously. Exhibit A shown below is the most common, but Exhibit B is not hard to reproduce under the same circumstances. Exhibits C and D are also fairly common, and though not directly SQL-related, suggest incomplete or corrupted recordsets. Sometimes one window will successfully load the page; sometimes neither will. Never do both windows successfully load the page simultaneously. I would appreciate any suggestions.[...]" Here is our problem : We the are working with Zope (mxODBCZopeDA 1.00 for connection) mxODBC and SQL Server 2000 too. We use ZSQL methods to define sql. We have an application showing some complex informations (retrieving from several databases), but sometimes the following errors occur: 1: Fuction Sequence Error : here is the log : "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in: call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 36, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZArticlesOuvragesFactory, line 38, in createTitre Module Dupuis.GestionCatalogue.ZDomaineCatalogue.ZTitre, line 50, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Titre, line 34, in __init__ Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 39, in __init__ Module Dupuis.GestionCatalogue.ZDBAccessCatalogue.ZArticlesFinder, line 20, in getNumeroArticleByNum Module Shared.DC.ZRDB.DA, line 428, in __call__ Module Products.mxODBCZopeDA.ZopeDA, line 1059, in query Module Products.mxODBCZopeDA.ZopeDA, line 1000, in run_cursor_callback Module Products.mxODBCZopeDA.ZopeDA, line 707, in errorhandler InterfaceError: ('HY010', 0, '[Microsoft][ODBC SQL Server Driver]Function sequence error', 4198)" (getNumeroArticleByNum call a ZSQL method doing a simple select in a database) This happen when Zope has to retrieve severall informations in a database, but a connection is already open on this database. 2 : Connection Busy We also had: "Error Type: ProgrammingError Error Value: ('HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911)" That happen in the same circumstances. 3 : Wrong resultset - Empty resultset, no commit : We call a ZSQL to get information from some article (identified with a known number) This raise some exception (we verify number exists before to fetch information) "Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module Products.PythonScripts.PythonScript, line 315, in _exec Module Script (Python), line 13, in choixserie - - Line 13 Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 54, in getListeTitreByNum Module Dupuis.GestionCatalogue.DomaineCatalogue.SerieCommerciale, line 84, in _initTitres Module Dupuis.GestionCatalogue.DomaineCatalogue.ArticlesOuvragesFactory, line 35, in getArticleWithCorrectType Module Dupuis.GestionCatalogue.DomaineCatalogue.Article, line 282, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 604, in isUnTitre Module Dupuis.GestionCatalogue.AbstractDBAccessCatalogue.ArticleDBGateway, line 712, in _getParams AbstractCatalogueDBException: Article1118320:Article1118320Inexistant" Wich means "no article with number 1118320", but this article exists!! We also have no error message, but query return an empty resultset (should not be) and finally, most important, when inserting some data in a database, it doesn t show any error, but the data isn't inserted (like if transaction didn t ends by a commit)... What we did : With all those problems, we had no other choice than supress ZSQL methods.... So we created a internal product for zope doing the following things : - open a new connection - writing sql with given parameters - getting results - closing connection With this, we can avoid all problems we got. But with this solution, we don t have transaction and rollback possibilities. It s the only thing we found. Obviously, this his solution can be temporary, but not definitive. Other solution? Is there something else we could do? Like change something in the mxODBC driver to avoid all those problems? Best regards Sebastien Walgraffe From mal at lemburg.com Tue Oct 21 19:11:58 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F955ACE.2020503@lemburg.com> S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From walgraffe at dupuis.com Wed Oct 22 11:36:24 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: <3F955ACE.2020503@lemburg.com> Message-ID: Thanks for your response (so fast ^^) But this seems not to be a solution for us because : - We already used a pool size of 1 (in fact we installed new version of MxODBC hoping that pool size would change those problems, but it didn t change...) - We don t have connection errors only on update followed by select... We also have the same problems on selects (loading a lot of rows from different tables for example) We have a single physical connection, but when we think at it, we don t understand how will parralelism could with different users, because if it works like a queue, other users ll have timeouts or errors Example : User A ask a huge amount of data from DB, requiring many queries User B ask a simple information -> Connection open for User A -> Transaction for A blocking the physical connection for B -> Timout or error for B Any clue? Thanks in advance Sebastien Walgraffe -----Message d'origine----- De : M.-A. Lemburg [mailto:mal@lemburg.com] Envoy? : mardi 21 octobre 2003 18:12 ? : S?bastien Walgraffe Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! S?bastien Walgraffe wrote: > Hello, > after seeing the same case described in the mailing list archives from > september, I decided to post my problem too, because it seems really similar > > Reminder : > nwingfield at che-llp.com wrote : > "We are having trouble with mxODBC and Microsoft SQL Server 2000, when two > or more people run the same query at the same time. This trouble is easily > reproduced by opening two browser windows and loading a page that executes > a lengthy SQL query in both windows simultaneously. Exhibit A shown below > is the most common, but Exhibit B is not hard to reproduce under the same > circumstances. Exhibits C and D are also fairly common, and though not > directly SQL-related, suggest incomplete or corrupted recordsets. > Sometimes one window will successfully load the page; sometimes neither > will. Never do both windows successfully load the page simultaneously. I > would appreciate any suggestions.[...]" > > Here is our problem : > > ... > > What we did : > > With all those problems, we had no other choice than supress ZSQL > methods.... > > So we created a internal product for zope doing the following things : > - open a new connection > - writing sql with given parameters > - getting results > - closing connection > With this, we can avoid all problems we got. > > But with this solution, we don t have transaction and rollback > possibilities. It s the only thing we found. > Obviously, this his solution can be temporary, but not definitive. > > Other solution? > > Is there something else we could do? Like change something in the mxODBC > driver to avoid all those problems? I can only guess, but since your problems sound very similar, I assume that the cause is the same... These problems generally occur if you are using two or more Z SQL Methods in a way that let's the SQL Server block, e.g. you have an UPDATE statement followed by SELECT statement in the same Zope transaction, but only if the Z SQL Methods use two or more mxODBC Zope DA connections or a single mxODBC Zope DA connection configured to use a connection pool of size 2 or more. The reason is that mxODBC Zope DA tries to make use of the available resources as best as possible... E.g. lets say you have configured two Z SQL Methods using a single mxODBC Zope DA connection with pool size 2. In the above example, the DA would give connection 1 to the first Z SQL Method and connection 2 to the second. Since the two connections are each using their own physical connection and thus their own database transaction context, SQL Server issues a row lock for the UPDATEd rows and let's the SELECT in the second method wait for that lock. Unfortunately, both Z SQL Methods are used within a single Zope transaction, so you get all kinds of weird errors from SQL Server. Other DAs like e.g. the ZODBC DA don't use connection pooling, so you'll never see the problem there... but then they don't even give you a chance to make use of parallelism which can often be put to good use, esp. in read-only cases. The solution to the problem is to either set the pool size in the used mxODBC Zope DA connection to 1 (all Z SQL Methods using this connection will then work with the same physical connection) or to code your application in a Z SQL Method independent way that makes sure that both statements are executed on the same physical connection. Hope that helps, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 21 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at lemburg.com Wed Oct 22 14:55:28 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:37 2006 Subject: [egenix-users] mxODBC and Microsoft SQL Server 2000 : An other case! In-Reply-To: References: Message-ID: <3F967030.6030608@lemburg.com> S?bastien Walgraffe wrote: > Thanks for your response (so fast ^^) > > But this seems not to be a solution for us because : > > - We already used a pool size of 1 (in fact we installed new version of > MxODBC hoping that pool size would change those problems, but it didn t > change...) > - We don t have connection errors only on update followed by select... We > also have the same problems on selects (loading a lot of rows from different > tables for example) > > We have a single physical connection, but when we think at it, we don t > understand how will parralelism could with different users, because if it > works like a queue, other users ll have timeouts or errors > Example : > User A ask a huge amount of data from DB, requiring many queries > User B ask a simple information > -> > Connection open for User A > -> > Transaction for A blocking the physical connection for B > -> > Timout or error for B > > Any clue? The problem you describe is a little different than the one I tried to explain in my previous mail: The Zope DA manages the connection using two layers: 1. a logical one per Zope Connection object 2. a physical one which pools the physical database connections If you set pool size to 1, only one physical connection is opened for the database. Now in your situation (and probably others as well, which is why I'm writing this here), Zope thread 1 will ask for a physical connection and receive it from the pool manager. Zope will then start a transaction on that connection, locking it against use by any other Zope thread. Once the transaction is started, Z SQL Methods in that Zope thread can happily use the connection for e.g. long running SELECTs. Since Zope is typcially running using more than one thread, a second request to the Zope server will usually result in Zope thread 2 to receive this request. Zope thread 2 will then ask the DA for a connection and it will return the same physical connection object as it did for Zope thread 1 (it will of course try to find a non-used one, but falls back to passing back a currently busy one). This is where you run into the blocking problem: Zope will try to start a transaction on this physical connection in Zope thread 2 as well. However, the transaction is already in use by Zope thread 1 and the physical connection managed by the Zope DA will wait for Zope thread 1 to release the lock before it lets Zope thread 2 use it for its own purposes. The solution to this problem is easy: set pool size to at least the number of threads you expect Zope to be running. However, keep in mind that when setting the pool size to more than 1, you can run into the locking problem I mentioned in my previous mail. This is only solvable by using a program design that carefully pays attention to row level locking and transaction isolation in the database server and can vary between database backends. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 22 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From sabaini at telbiomed.at Wed Oct 22 20:40:33 2003 From: sabaini at telbiomed.at (Peter Sabaini) Date: Fri Mar 31 16:33:38 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load Message-ID: <3F96C111.1010002@telbiomed.at> Hello list, I'm getting an error running ZmxODBCDA under load -- 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 This happens quite deterministically any time I put some load on the server. Has anyone else seen this? Help much appreciated... System: * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) * MDAC 2.8 * Windows 2k SP4 * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 * Testbox is a PIII with 320Mb RAM * ZmxODBCDA is configured with 8 pooled connections, all other settings are default I've created a test case that (hopefully) makes this reproducible. The Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml In the test app I generate 10 DB queries HTTP request (the query is 'select count(*) from sysobjects', which is around 1700 on my DB). Then I hit the testskript with something like ab2 -n 2000 -c 20 -k -v2 \ -C _ZopeId=35375730A1CCu3EFDTs \ http:///zmxodbcda_tst/select_test_py Ie. 20 concurrent requests, 2000 total Thanks a lot for any feedback, peter sabaini. -- ========================================================================= Peter Sabaini ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine Grottenhofstrasse 3, A-8053 Graz T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at H: http://telbiomed.at, IM: chrome___@jabber.at ----------------------------- Meet us at the MEDICA 2003 in Duesseldorf 19. - 22. Nov. Hall 14 Stand D15 From nwingfield at che-llp.com Wed Oct 22 14:57:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Mar 31 16:33:38 2006 Subject: [egenix-users] ZmxODBCDA, MS-SQL under load In-Reply-To: <3F96C111.1010002@telbiomed.at> Message-ID: Peter, You're certainly not the only one. This is a problem specific to SQL Server. I am currently working on an experimental workaround, which involves removing Z SQL methods altogether. I'll post to the board whatever I find out, hopefully within the next day or two. Nathaniel egenix-users-bounces@lists.egenix.com wrote on 10/22/2003 01:40:33 PM: > > Hello list, > > I'm getting an error running ZmxODBCDA under load -- > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > > This happens quite deterministically any time I put some load on the server. > > Has anyone else seen this? Help much appreciated... > > > System: > > * MS SQL 2000 (Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 > 2002 14:22:05 Copyright (c) 1988-2003 Microsoft Corporation Standard > Edition on Windows NT 5.0 (Build 2195: Service Pack 4)) > > * MDAC 2.8 > > * Windows 2k SP4 > > * Zope 2.6.1 binary w/ 4 threads, Python 2.1.3 > > * Testbox is a PIII with 320Mb RAM > > * ZmxODBCDA is configured with 8 pooled connections, all other settings > are default > > > I've created a test case that (hopefully) makes this reproducible. The > Zope testing app can be downloaded from http://sabaini.at/zmxodbcda_tst.xml > > In the test app I generate 10 DB queries HTTP request (the query is > 'select count(*) from sysobjects', which is around 1700 on my DB). Then > I hit the testskript with something like > > ab2 -n 2000 -c 20 -k -v2 \ > -C _ZopeId=35375730A1CCu3EFDTs \ > http:///zmxodbcda_tst/select_test_py > > > Ie. 20 concurrent requests, 2000 total > > > > Thanks a lot for any feedback, > > peter sabaini. > > > > > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Wed Oct 22 18:48:21 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Mar 31 16:33:38 2006 Subject: [egenix-users] SQL Server: Experimental Solution Message-ID: We have been experiencing many problems with mxODBC and SQL Server, similar to those expressed by others list members (S?bastien Walgraffe, Peter Sabaini). The most common error is this: 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt', 5911 By eliminating Z SQL methods, we seem to have remedied this error, though only time will tell. We have replaced Z SQL methods with the External Method that I am including below. Another problem that we experienced was that when the connection pool was set greater than one, pages executing multiple SQL statements could lock up. This is because each SQL method was potentially executed on a different connection. Rather than limit ourselves to a connection pool of one, we incorporated some logic into the aforementioned method to maintain one connection per request. If the locking isn't a problem for you, you can strip the extra logic out of this method. Let me know if this helps. def executeSQL(sstmt,odbcObject,REQUEST,max_rows=None): """ DESCRIPTION: This method will execute an SQL statement using the mxODBC Connector provided. For the duration of the request, it ensures that only one connection is used. This is done in order to prevent weird hard locks on SQL Server. PARAMETERS: sstmt: well-formed SQL to be executed (don't forget the semicolon!) odbcObject: the mxODBC data connector to be used REQUEST: context.REQUEST max_rows: maximum rows to retrieve """ REQUEST_KEY = 'DB_CONNECTION' + '_'.join(odbcObject.getPhysicalPath()) sstmt = sstmt.strip() if not hasattr(REQUEST,REQUEST_KEY): REQUEST[REQUEST_KEY] = odbcObject.get_connection() conn = REQUEST[REQUEST_KEY] results = conn.query(sstmt,max_rows=max_rows) rs = Results(results) return rs From mal at lemburg.com Thu Oct 23 14:12:09 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:38 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F97B789.2030600@lemburg.com> nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nwingfield at che-llp.com Thu Oct 23 11:10:53 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Mar 31 16:33:38 2006 Subject: [egenix-users] SQL Server: Experimental Solution In-Reply-To: <3F97D1B6.9040706@telbiomed.at> Message-ID: Peter Sabaini wrote on 10/23/2003 09:03:50 AM: > How does this play along with the Zope transaction machinery? I'm not sure of the answer to this question. Does anyone on the list know? I'm a bit worried about this myself. > Also, since we have tons of ZSQL Methods around, this is a bit > impractical for us :-( You don't need to tell me about lots of Z SQL methods! I had to convert 153 of them to Python scripts, each calling my executeSQL external method. If you would like more details on how I achieved this with minimal effort, I will be glad to explain more. Nathaniel > > nwingfield@che-llp.com wrote: > > We have been experiencing many problems with mxODBC and SQL Server, similar > > to those expressed by others list members (S?bastien Walgraffe, Peter > > Sabaini). The most common error is this: > > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > > results for another hstmt', 5911 > > By eliminating Z SQL methods, we seem to have remedied this error, though > > only time will tell. We have replaced Z SQL methods with the External > > Method that I am including below. Another problem that we experienced was > > that when the connection pool was set greater than one, pages executing > > multiple SQL statements could lock up. This is because each SQL method was > > potentially executed on a different connection. Rather than limit > > ourselves to a connection pool of one, we incorporated some logic into the > > aforementioned method to maintain one connection per request. If the > > locking isn't a problem for you, you can strip the extra logic out of this > > method. > > > > Let me know if this helps. > > > > -- > ========================================================================= > Peter Sabaini > ARC Seibersdorf research GmbH, Biosignal Processing and Telemedicine > Grottenhofstrasse 3, A-8053 Graz > T: +43 (0)316 265 158, F: +43 (0)316 265 155; E: sabaini@telbiomed.at > H: http://telbiomed.at, IM: chrome___@jabber.at > ----------------------------- > Meet us at the MEDICA 2003 in Duesseldorf > 19. - 22. Nov. Hall 14 Stand D15 > > > > > From walgraffe at dupuis.com Mon Oct 27 11:28:53 2003 From: walgraffe at dupuis.com (=?iso-8859-1?Q?S=E9bastien_Walgraffe?=) Date: Fri Mar 31 16:33:38 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution Message-ID: Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From nwingfield at che-llp.com Mon Oct 27 09:45:24 2003 From: nwingfield at che-llp.com (nwingfield@che-llp.com) Date: Fri Mar 31 16:33:38 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: Message-ID: Sure you do: self.REQUEST in a product is the same as context.REQUEST in a Python Script or here.REQUEST in a Zope Page Template. Nathaniel S?bastien Walgraffe Sent by: cc egenix-users-boun ces@lists.egenix. Subject com TR: [egenix-users] SQL Server: Experimental Solution 10/27/2003 05:28 AM Hello We can t use the solution proposed by nwingfield@che-llp.com. Because we call ZSQL from a product and we don t have at this place the REQUEST. With the modifications that you planned in Vesrion 1.0.7, do you think that it will work if we call ZSQL without REQUEST parameter ? Thanks Sebastien Walgraffe -----Message d'origine----- De : egenix-users-bounces@lists.egenix.com [mailto:egenix-users-bounces@lists.egenix.com]De la part de M.-A. Lemburg Envoy? : jeudi 23 octobre 2003 13:12 ? : nwingfield@che-llp.com Cc : egenix-users@lists.egenix.com Objet : Re: [egenix-users] SQL Server: Experimental Solution nwingfield@che-llp.com wrote: > We have been experiencing many problems with mxODBC and SQL Server, similar > to those expressed by others list members (S?bastien Walgraffe, Peter > Sabaini). The most common error is this: > 'HY000', 0, '[Microsoft][ODBC SQL Server Driver]Connection is busy with > results for another hstmt', 5911 > By eliminating Z SQL methods, we seem to have remedied this error, though > only time will tell. We have replaced Z SQL methods with the External > Method that I am including below. Another problem that we experienced was > that when the connection pool was set greater than one, pages executing > multiple SQL statements could lock up. This is because each SQL method was > potentially executed on a different connection. Rather than limit > ourselves to a connection pool of one, we incorporated some logic into the > aforementioned method to maintain one connection per request. If the > locking isn't a problem for you, you can strip the extra logic out of this > method. > > Let me know if this helps. If this does solve your problems, we'll add a connection option to the Zope DA which will let you switch on/off this behaviour (grouping connections per request) for version 1.0.7. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 23 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users _______________________________________________________________________ eGenix.com User Mailing List http://www.egenix.com/ http://lists.egenix.com/mailman/listinfo/egenix-users From mal at lemburg.com Tue Oct 28 13:39:20 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:38 2006 Subject: TR: [egenix-users] SQL Server: Experimental Solution In-Reply-To: References: Message-ID: <3F9E6378.3040300@lemburg.com> S?bastien Walgraffe wrote: > Hello > > We can t use the solution proposed by nwingfield@che-llp.com. Because we > call ZSQL from a product and we don t have at this place the REQUEST. > With the modifications that you planned in Vesrion 1.0.7, do you think that > it will work if we call ZSQL without REQUEST parameter ? We'll try to come up with a solution that works without you having to change the code. Each request or thread will then (optionally) use just one connection, so you can still benefit from having multiple requests running parallel, but you don't run into deadlock problems within one request. Note that a long running update query can still block other requests. There's no way around exclusive locks set by the database. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From F.Baart at SFK.NL Wed Oct 29 11:25:47 2003 From: F.Baart at SFK.NL (F.Baart@SFK.NL) Date: Fri Mar 31 16:33:38 2006 Subject: [egenix-users] problem connecting to mysql Message-ID: Dear somebody, I'm having some problems connecting a windows zope machine to our linux mysql machine. I did the following: I've added a new user to mysql. I've added a mysql connection to odbc in the windows configuration panel (without specifying the username and password). I've added a egenix mxODBC database connection in Zope. I specified the connection string as "DSN=mysql_odbc_connection;UID=user;PWD=password" When I open the connection it seems like the uid and pwd aren't supplied to mysql. I get an error that reads: """ Connection string: DSN=mysql_odbc_connection; UID=user; PWD=password Connection pool entry: 0 Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) """ When I use the ZODBC connection and supply "mysql_odbc_connection user password" it works nicely. Also when I add the username and password to the mysql connection it works. But I'd rather supply the username and password in the connection string. I hope somebody can help me to use the egenix odbc connection to mysql. Thanks, Fedor Baart SFK From mal at lemburg.com Wed Oct 29 11:31:24 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:38 2006 Subject: [egenix-users] problem connecting to mysql In-Reply-To: References: Message-ID: <3F9F96FC.5090600@lemburg.com> F.Baart@SFK.NL wrote: > Dear somebody, > > I'm having some problems connecting a windows zope machine to our linux > mysql machine. > > I did the following: > > I've added a new user to mysql. > I've added a mysql connection to odbc in the windows configuration panel > (without specifying the username and password). > I've added a egenix mxODBC database connection in Zope. > I specified the connection string as > "DSN=mysql_odbc_connection;UID=user;PWD=password" > > When I open the connection it seems like the uid and pwd aren't supplied to > mysql. I get an error that reads: > > """ > Connection string: DSN=mysql_odbc_connection; > UID=user; > PWD=password > > > Connection pool entry: 0 > > Error message: ('HY000', 1045, "[MySQL][ODBC 3.51 Driver]Access deni > ed for user: 'ODBC@zopeserver' (Using password: NO)", 7935) > """ Try putting the connection string on one line, ie. DSN=mysql_odbc_connection;UID=user;PWD=password Some ODBC drivers are very picky about line breaks in the DSN and MySQL's seems to be one of them. > When I use the ZODBC connection and supply "mysql_odbc_connection user > password" it works nicely. > Also when I add the username and password to the mysql connection it works. > But I'd rather supply the username and password in the connection string. > I hope somebody can help me to use the egenix odbc connection to mysql. > > Thanks, > > Fedor Baart > SFK > > > > > _______________________________________________________________________ > eGenix.com User Mailing List http://www.egenix.com/ > http://lists.egenix.com/mailman/listinfo/egenix-users -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 29 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From Kelly.Prendergast at noaa.gov Thu Oct 30 15:00:18 2003 From: Kelly.Prendergast at noaa.gov (Kelly Prendergast) Date: Fri Mar 31 16:33:38 2006 Subject: [egenix-users] writing binary files Message-ID: <3FA189F2.4C928EC7@noaa.gov> Python newbie here, I am retrieving data from SQL Server that is stored as a 144 byte binary. If I understand the Phyton documentation, the data is returned a ptyhon string type What is the best way to write this string to a binary file? Thanks, Kelly From mal at lemburg.com Fri Oct 31 10:02:47 2003 From: mal at lemburg.com (M.-A. Lemburg) Date: Fri Mar 31 16:33:38 2006 Subject: [egenix-users] writing binary files In-Reply-To: <3FA189F2.4C928EC7@noaa.gov> References: <3FA189F2.4C928EC7@noaa.gov> Message-ID: <3FA22537.2050903@lemburg.com> Kelly Prendergast wrote: > Python newbie here, > > I am retrieving data from SQL Server that is stored as a 144 byte > binary. If I understand the Phyton documentation, the data is returned a > ptyhon string type > What is the best way to write this string to a binary file? The short version: open('filename.bin', 'wb').write(stringdata) For a longer version, I'd suggest you have a look at one of the nice Python books out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Oct 31 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::